API Reference / Adding Backends

Table of Contents [Sticky]

Introduction

The backends represent mechanisms to execute the image mapping in the correction stage. They usually offer efficient ways to perform the computations, such as OpenCL, CPU SIMD, CUDA, and others. It usually involves the following steps:

Define a custom allocator to store the image buffers.
Define the IRuntimeSettings specific for the backend.
Code the algorithm kernels, inheriting from IUndistort.

This recalls the following interfaces:

In this section, we will explore how to add new execution backends to the RidgeRun Video Stabilization Library, going through the aforementioned steps.

Define a Custom Allocator

The RidgeRun Video Stabilization Library is extensible and can handle different execution backends. These backends may require different allocation techniques, such as alignment, physically contiguous, non-paged, or device memory. The IAllocator helps implement new allocators required for the backends. Moreover, they safeguard the Image class to avoid mixing objects whose memory buffers are incompatible.

This recalls the following interface:

Image Interface

In the Image<F<T>,A> class, the A template argument helps to introduce the allocator implementation. Moreover, the allocators possess methods to ensure a good alignment when storing images, such as the BestPitch method.

To implement a new allocator, the following actions must be performed:

Define the Alloc method

It must receive a size in bytes, and alignment in bits (optional) and return a pointer (uint8_t *).

A possible implementation is the following:

Ptr CustomAllocator::Alloc(const size_t length, const size_t alignment) {
  return new Ptr[length];
}

The method can enforce a fixed alignment.

Define the Dealloc method

It must receive the pointer and deallocate it.

void CustomAllocator::Dealloc(Ptr pointer) {
  delete[] pointer;
}

Define the BestPitch method

It receives a length in bytes and must return a valid length according to the alignment. For instance, if the length is 3 bytes and the alignment is 32-bit, the returned size is 4 bytes.

A possible implementation:

size_t BestPitch(const size_t length) const {
  static constexpr int kAlignBytes = 4;
  size_t mod = length % kAlignBytes;
  return mod == 0 ? length : length + (kAlignBytes - mod);
}

Define the Runtime Settings

The runtime settings define configurations of the execution backend, such as context, the number of local size workers, the number of threads, and the work queue (or stream). Some backends, like the OpenCV, do not require any special configuration. The runtime settings are used by the IUndistort adapters according to their implementation. An IUndistort adapter will meet a specific backend and, therefore, a set of settings defined for that specific backend, as illustrated in the following diagram.

Undistort Interface

A possible implementation for a CUDA Runtime Settings:

 struct CudaRuntimeSettings : public IRuntimeSettings {
   cudaStream_t stream = (cudaStream_t)(0);
   dim3 blockdims = dim3(32, 32);
   virtual ~CudaRuntimeSettings() = default;
 };

Define the Algorithm Kernels

The final step for the backend implementation is to define the IUndistort adapter. It contains the Apply method that performs the operation on the image. We encourage the use of strong type checking types after the Apply. Please, follow this example as a reference:

 class CustomFishEye : public IUndistort {
  public:
   // Define the compatible format
   using FormatType = RGBA<uint8_t>;
   // Define the image type: notice that we set the format and a custom allocator (based on above)
   using ImageType = Image<FormatType, CustomAllocator>;
  
   // Constructor: initialises the backend based on the settings (if passed)
   //              constructors can also create their own settings using defaults
   explicit FishEyeOpenCV(const std::shared_ptr<IRuntimeSettings> settings);

   // Define the interface Apply method. See below how checking are done
   RuntimeError Apply(std::shared_ptr<IImage> outframe,
                      const std::shared_ptr<IImage> inframe,
                      const Quaternion<double> &rotation,
                      const double fov_scale) override;
  
   // Sets new Runtime Settings. It may require a reinitialisation of the backend
   RuntimeError SetRuntimeSettings(const std::shared_ptr<IRuntimeSettings> settings) override;

   // Creates and returns a copy of the runtime settings from this instance
   std::shared_ptr<IRuntimeSettings> GetRuntimeSettings() override;
  
   // Virtual destructor that finalises the backend
   virtual ~CustomFishEye();
  
  private:
   // Custom allocator compatible with the backend. For intermediate buffers
   CustomAllocator allocator_;
  
   // Custom strong-typed Apply method. The virtual Apply must call this after checkings
   RuntimeError Apply(ImageType &outframe, const ImageType &inframe,  // NOLINT
                      const Quaternion<double> &rotation,
                      const double fov_scale);
 };
 
 RuntimeError CustomFishEye::Apply(std::shared_ptr<IImage> outframe,
                                   const std::shared_ptr<IImage> inframe,
                                   const Quaternion<double> &rotation,
                                   const double fov_scale) {
  // Check incoming images
  std::shared_ptr<ImageType> inframeptr = std::dynamic_pointer_cast<ImageType>(inframe);
  std::shared_ptr<ImageType> outframeptr = std::dynamic_pointer_cast<ImageType>(outframe);

  if (nullptr == inframeptr || nullptr == outframeptr) {
    return RuntimeError{RuntimeError::IncompatibleParameters,
                        "The images must be Host-Allocated RGBA images"};
  }

  // Check passed, use the apply implementation
  return this->Apply(*outframeptr, *inframeptr, rotation, fov_scale);
 }

Register the new implementations

After that, you can register the new implementations as follows:

The undistort in the UndistortAlgorithms enum available in the iundistort.hpp file.
The allocators in the Allocators enum available in iallocator.hpp file.
The undistort in the factory in the iundistort.cpp file.
The allocators in the factory in the iallocator.cpp file.